Solid-state imaging elements, such as CCD image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors used in video cameras, digital still cameras, and such, accumulate electrical charges corresponding to an amount of incident light, and perform photoelectric conversion to output electrical signals corresponding to the accumulated electrical charge. However, there is a limit to the amount of electrical charge that may be accumulated in the photoelectrical conversion elements, and if an amount of light over a certain amount is received, the accumulated electrical charge amount approaches a saturated level, and if a photographic subject region has a brightness that is over a certain amount, a so-called whiteout condition occurs in those regions where a luminosity level has become saturated.
In order to prevent this kind of phenomenon, an electrical charge accumulation period for the photoelectric conversion elements is controlled corresponding to changes in external light and similar, exposure periods are adjusted, and processing is performed to control sensitivity to an optimum value. For example, for a bright photographic subject, by cutting a shutter at high speed, the exposure period is shortened, and the photoelectric accumulation period for the photoelectric conversion elements is shortened to cause an electrical signal to be output before the accumulated charge reaches the saturation level. With this kind of processing, images in which their gradation is correctly reproduced corresponding to the photographic subject may be output.
However, for the photographing of photographic subjects with both bright and dark areas, if the shutter is cut at a high speed, the exposure period is not long enough for the dark areas, which results in S/N degradation and loss of image quality. For photographic images of photographic subjects with both bright and dark areas, it is necessary to realize a high S/N with a long exposure period for pixels on the image sensor which have a small amount of incident light, and to perform processing to avoid saturation for pixels which have a large amount of incident light in order to correctly reproduce both the bright areas and the dark areas.
As a method to realize this kind of processing, a method to combine multiple images photographed consecutively with different exposure times is known. That is to say, this is a method to generate one image in which long exposure images and short exposure images are consecutively photographed individually, and by a combination processing in which the long exposure images are used for the dark image regions, and the short exposure images are used for the bright image regions which would have whiteout in the long exposure images. In this way, by combining multiple, different exposure images, images with no whiteout and a wide dynamic range may be obtained.
For example, PTL 1 (Japanese Unexamined Patent Application Publication No. 2008-99158) discloses a configuration to obtain images with a wide dynamic range by combining multiple images with different exposure amounts. This processing will be described with reference to FIG. 1. Regarding shooting moving images, an imaging device, for example, outputs image data with two different exposure times within a video rate (30-60 fps). Also, regarding shooting still images, image data with two different exposure times is generated and output. FIG. 1 is a diagram describing properties of images with two different exposure times generated by the imaging device (long exposure image and short exposure image). The horizontal axis is time (t), and the vertical axis is an accumulated electrical charge amount (e) for a light-receiving photodiode (PD) that configures a photoelectric conversion element corresponding to one pixel of a solid-state imaging element.
For example, when the incident light amount of the light-receiving photodiode (PD) is large, that is to say when handling a bright photographic subject, as represented by a high luminosity region 11 as illustrated in FIG. 1, the electrical charge accumulation amount rapidly increases along with the elapsed time. In contrast, when the incident light amount of the light-receiving photodiode (PD) is small, that is to say when handling a dark photographic subject, as represented by a low luminosity region 12 as illustrated in FIG. 1, the electrical charge accumulation amount rises mildly along with the elapsed time.
Timings t0 through t3 are equivalent to an exposure time TL for obtaining long exposure images. For the line representing the low luminosity region 12 as this long exposure time TL, the electrical charge accumulation amount does not reach a saturation level at the timing t3 (non-saturation point Py), and a correct gradation expression may be obtained by using an electrical signal obtained based on this electrical charge accumulation amount (Sa) to determine a gradation level of the pixel.
However, it is obvious that the electrical charge accumulation amount for the line representing the high luminosity region 11 has already reached the saturation level (saturation point Px) before the timing t3. Therefore, from this kind of high luminosity region 11, only pixel values corresponding to electrical signals at a saturation level may be obtained from the long exposure images, and as a result pixels will whiteout.
Thus, at this kind of high luminosity region 11, for the time before leading up to the timing t3, for example the timing t1 illustrated in the diagram (electrical discharge start point P1), first the electrical charge from the light-receiving photodiode (PD) will be discharged. The electrical charge discharged is not the entire electrical charge accumulated in the light-receiving photodiode (PD), but only the intermediate voltage retaining level that is controllable for the photodiode (PD). After this electrical charge discharge processing, a short exposure executes again, which is an exposure time TS (t2 through t3). That is to say, a short exposure will be performed during a period from a short exposure start point P2 to a short exposure end point P3 as illustrated in the figure. An electrical charge accumulation amount (Sb) may be obtained by this short exposure, and the pixel gradation level is determined based on the electrical signal obtained based on this electrical charge accumulation amount (Sb).
Further, when determining pixel values based on the electrical signal based on the electrical charge accumulation amount (Sa) obtained by the long exposure for the low luminosity region 12 and the electrical signal based on the electrical signal based on the electrical charge accumulation amount (Sb) obtained by the short exposure for a high luminosity region 251, an electrical signal output value is calculated corresponding to an estimated electrical charge accumulation amount when equal time exposure is performed or this estimated electrical charge accumulation amount, and a pixel value level is determined based on the calculation result.
In this way, by combining short exposure images and long exposure images, images with no whiteout that have a wide dynamic range may be obtained.
Further, PTL 2 (Japanese Unexamined Patent Application Publication No. 2000-50151) discloses a configuration similar to the configuration described in PTL 1 in which multiple images with different exposure amounts are photographed, and during the combination processing, the multiple images with different exposure amounts are compared, and a pixel region that contains movement is identified, and a correction is performed to control the generation of false color that accompanies the combination.
However, the configurations described in the PTLs 1 and 2 described previously have to perform processing to photograph and combine long exposure images and short exposure images separately at some point. To perform this processing, there has to be enough frame memory to store at least one image worth of data, which has been a problem that has led to increased costs. Also, the DSP (Digital Signal Processor) that performs the image processing has to perform processing in which data for two images with different exposure times are input. As a result, this required a processor with advanced functionality, and this point has also been a problem which has led to increased camera costs.
Also, as the photographing of at least two images has to be executed, there is a problem in which too much time has to be taken for the photographing time and the processing time. Also, when the photographic subjects move and movement occurs during the period when photographing the two images, this causes a problem in that good image combination cannot be performed, and this lowers the quality of the combined image output.